Papers by William F. Arnold
Chatbot Arena Estimate: towards a generalized performance benchmark for LLM capabilities (2025.naacl-industry)
Copied to clipboard
Lucas Spangher, Tianle Li, William F. Arnold, Nick Masiewicki, Xerxes Dotiwalla, Rama Kumar Pasumarthi, Peter Grabowski, Eugene Ie, Daniel Gruhl
| Challenge: | Existing benchmark aggregation methods, such as Elo-based systems, can be resource-intensive, public facing, and time-consuming. |
| Approach: | They propose a framework for aggregating performance across diverse benchmarks that generates a “Goodness” and a ‘Fastness” score. |
| Outcome: | The proposed framework achieves higher Pearson correlation with Chatbot Arena Elo scores than MMLU’s correlation with chatbot Arena scores, validating its reliability for real-world LLM evaluation. |
RLHF Algorithms Ranked: An Extensive Evaluation Across Diverse Tasks, Rewards, and Hyperparameters (2025.emnlp-industry)
Copied to clipboard
Lucas Spangher, Rama Kumar Pasumarthi, Nick Masiewicki, William F. Arnold, Aditi Kaushal, Dale Johnson, Peter Grabowski, Eugene Ie
| Challenge: | Proximal Policy Optimization (PPO) has fallen out of favor for Large Language Models (LLMs), but its complexity and inefficiency have spurred the investigation of simpler alternatives. |
| Approach: | They evaluate 17 RLHF algorithms on two benchmarks, OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness. |
| Outcome: | The proposed methods are based on OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness benchmarks with two different reward models and a Rules based reward model. |